Scenario
Our services on Google Cloud (once called GCP) use Cloud NAT, and recently we've been experiencing intermittent issues that seem to be caused by dropped packets in the NAT. The current solution is to increase the number of external NAT IPs on GCP. However, we cannot arbitrarily add NAT IPs because the destination has whitelisted our current IPs. How should we address this issue?
Operation
There are two common scenarios
Scenario 1: Out of Resources
You can reduce your port usage by:
- Disabling Endpoint-Independent Mapping.
- Enabling dynamic port allocation.
- Ensuring the minimum use of ports.
Scenario 2: Endpoint-Independent Conflict
You can disable Endpoint-Independent Mapping or increase the minimum number of ports per VM.
Setting the minimum number of ports per VM instance to 30,000 means that each backend VM will have at least this number of ports opened. This number also affects the number of VMs that a single NAT IP can support.
According to the official documentation, the calculation is roughly as follows (the final numbers are rounded down):
⌊(1 NAT IP) × (64,512 ports per IP) / (64 ports per VM)⌋ = 1,008 VMs
⌊(2 NAT IPs) × (64,512 ports per IP) / (64 ports per VM)⌋ = 2,016 VMs
⌊(1 NAT IP) × (64,512 ports per IP) / (4,096 ports per VM)⌋ = 15 VMs
Impact of Adjusting Port Numbers:
If using static allocation, increasing the port count has no real impact on traffic. Decreasing it may disrupt existing NAT connections, requiring clients to re-establish TCP connections.
If using dynamic allocation, whenever the configuration changes (whether increasing or decreasing), the number of ports allocated to VMs will temporarily reset to the minimum, which may affect some traffic. Alternatively, you could reduce port usage.
References
[1] https://cloud.google.com/nat/docs/troubleshooting#insufficient-ports
[2] https://cloud.google.com/nat/docs/troubleshooting#endpoint-independent-conflict
[3] https://cloud.google.com/nat/docs/ports-and-addresses#port-reservation-examples
[4] https://cloud.google.com/nat/docs/ports-and-addresses#specs-increasing
[5] https://cloud.google.com/nat/docs/ports-and-addresses#specs-reducing
[6] https://cloud.google.com/nat/docs/troubleshooting#reduce-ports
